Tools help us accomplish our daily tasks. A parent uses an electric or gas range, pots and pans, and spoons, spatulas and other implements to cook a meal for his or her family. A carpenter uses planes, saws, hammers and levels to cut pieces of wood to size and fasten them in. A mechanic uses wrenches, gear pullers, jacks, lifts and other tools to disassemble and reassembly a car engine. A gardener uses rakes, hoes, clippers and other implements to help nurture plants.
Just as in the real world, tools are also important in virtual realty, augmented reality and computer graphic simulations. Therefore, much work has been done in the past to develop efficient ways for a human user to control virtual tools displayed on a screen. In some cases, nearly the same controls a human user will operate in the real world can be used to control a computer graphics simulation of the same tool.
For example, an aircraft simulator used to train pilots often resembles nearly exactly a cockpit of an airplane. Large display screens are used to display what a pilot would see outside the window as if her or she were flying an actual airplane. The pilot in training operates the same controls found in a real cockpit to control the virtual aircraft simulation.
As another example, surgeons are sometimes trained using computer simulators that simulate the way the human body responds to surgical procedures. In such surgical simulations, a virtual scalpel may be used to cut tissue, a virtual suction device may be used to suction away blood and debris, and a virtual suture may be used to sew up the simulated wound. Such simulations allow surgeons to try new surgical techniques—oftentimes using real world controls that very closely resemble the implements they will use in the operating room—without risking the health and well being of actual human patients. Although there is no substitute for actual real world experience, such simulations can be useful as a first step toward training new surgeons.
Although designers of expensive aircraft, surgical and other simulators have the luxury of advanced, authentic controls and other input devices, designers of video game systems and other consumer based computer systems are often presented with a different set of challenges. For example, the conventional home video game platform, personal computer or handheld video game playing platform often has a relatively limited set of standard, general-purpose input devices the human player can use to manipulate video game action. Similarly, the computer keyboard and mouse of a conventional personal computer are generally used to control nearly all game functions of video games played on personal computers. Serious PC gamers will sometimes invest in a joystick peripheral device to allow more advanced directional control of certain games. However, such additional peripheral devices become impractical when gamers attempt to play games on portable platforms such as portable video gaming systems, cellular telephones with graphics capabilities, personal digital assistants, pocket PCs, and a host of other devices. In those contexts, the control interface that is available with the device out of the box is typically the one most players end up using for nearly all of their interactions. A challenge is to make maximum use of such input devices to create user-friendly, effective graphical user interfaces.
But while designers of video games, virtual and augmented reality based systems, and other virtual computer-generated environments may have certain limitations, they also have an additional degree of freedom. Just like in the real world, a character within a video game or other computer-generated environment may also use tools to accomplish certain virtual goals. But because computer generated environments are virtual, developers are not limited to real world tools. They can create new or fanciful tools not possible in the real world.
For example, the main character Link in Nintendo's highly successful Legend of Zelda video game series uses a sword, a musical instrument called an ocarina, a “windwaker” stick, a bow and arrow, a boomerang, a telescope, a fishing rod, and various other tools to accomplish certain results within the games. Some of these tools resemble real world objects, but within the game they often have magical properties. For example, Link can travel through time and otherwise magically manipulate his environment by playing certain songs on his ocarina.
Another interesting example is the “Poltergust 4000” vacuum cleaner strapped to the game character Luigi's back in Nintendo's renowned “Luigi's Mansion” video game. While this virtual vacuum cleaner has some resemblance to the kind of vacuum cleaner you clean your carpets with, it has amazing properties no real world vacuum cleaner would ever exhibit.
The human game player can operate Luigi's “Poltergust 4000” vacuum cleaner just like a real vacuum cleaner to “suck” (suction mode) or “blow” (exhaust mode). In the suction mode, the virtual vacuum cleaner sucks up objects within Luigi's virtual environment including but not limited to ghosts, fire spirits, water, ice cubes and balls. In the exhaust mode, the vacuum cleaner can exhaust some of these vacuumed-up items in fanciful ways. For example, if a fire spirit was the last item Luigi's vacuum cleaner sucked up, then operating Luigi's vacuum cleaner in the exhaust mode transforms the vacuum cleaner into a flame thrower. Similarly, if the last item the game player caused Luigi's vacuum player to suck up was water from a fountain, then Luigi's vacuum cleaner will in the exhaust mode act as a fire hose that emits a virtual water spray. This creative way of enhancing the functionality of an everyday appliance with magical or extraordinary properties has found great favor among millions of video game players throughout the world.
Despite the limitations in low cost video game and other consumer systems, there is an interesting relationship between the input devices the human game player uses to control a video game or other virtual environment, and the tools within the video game or other virtual environment used to accomplish objectives. In video games such as the Legend of Zelda and Luigi's Mansion, the human player uses buttons and/or joy sticks on a handheld controller (e.g., the “C stick”) to control the operation of the tools the game characters use within the game. For example, in the Luigi's Mansion game, the human player can control whether Luigi's virtual “Poltergust 4000” vacuum cleaner is in the suction mode, the exhaust mode, a flashlight mode, or no mode by manipulating buttons and/or joysticks on the Nintendo GameCube handheld controller. The handheld controller functions as a control panel for Luigi's virtual vacuum cleaner, allowing the human video game player to control where the “Poltergeist 4000” vacuum cleaner is aiming and what mode it is operating in. The same joysticks and buttons would be used to control direction, acceleration, turning radius and other attributes of a vehicle in a driving or flight simulator game.
Video game and computer graphics designers are constantly searching for new and innovative ways to make interaction with virtual, simulated and augmented reality worlds more efficient, cost-effective and enjoyable. Recently, touch screen based user interfaces have been introduce in the context of video game systems, banking terminals and other computer devices as a way to interact more efficiently with a graphical-based computer system. However, further improvements are possible and desirable.
The technology herein uses a 3D graphical computer interface to enhance the function of a physical object used to interact with a video game, computer simulation or any other graphical presentation. The object a human user holds in his or her hands may be relatively static or non-electronic, and yet may—through interaction with a specialized user interface provided via a multimedia (e.g., audio and/or graphical video) display presentation—appear to possess enhanced functional characteristics and properties. An ordinary handheld object may, through interaction with a computer graphical interface, appear, for example, to take on extraordinary or even magical virtual tool characteristics.
In one specific exemplary illustrative non-limiting implementation, the stylus used to interact with a touch screen based video game interface may, through simulation of enhanced functions on a computer graphics screen, appear to take on characteristics of a virtual vacuum or suction tool. In this exemplary illustrative non-limiting implementation, the stylus itself may be a conventional elongated stick-like implement such as a piece of inert hard plastic. Such a stylus in one exemplary illustrative non-limiting implementation has no internal cavity, no electronics, and substantially no mechanical function other than to indicate touching positions on a touch pad or touch screen. An exemplary illustrative non-limiting implementation of the technology herein uses computer graphics and computer-generated sound to animate interaction with such a conventional stylus to provide suction and exhaust properties of a virtual suction tool.
For example, unlike a real suction tool which may have relatively limited ability to pick up and put down real world objects in terms of dimensions, object characteristics and the like, the virtual suction tool provided by one exemplary illustrative non-limiting implementation of the technology herein can have fanciful object manipulation capabilities such as infinite capacity to suck up objects of all sorts. In a puzzle game for example, the virtual suction tool may be able to suck up any number of bubbles, blocks or other objects within the game. As the human user moves the tip of the virtual suction tool over the objects, each object may disappear in a way that creates an impression that the stylus is sucking the object up into an internal cavity or reservoir within the stylus. For example, as the object is being virtually suctioned, portions of the object closest to the tool may deform on the screen to make it appear as if the object has been partially sucked into the virtual vacuum tool. Once the stylus virtually contains the entire object that has been sucked up, the object may disappear from the display.
In a further exemplary non-limiting illustrative implementation, a noise, such as a sucking or whooshing noise, may accompany each capture of an object, so that the player feels as if the stylus has actually sucked the object from the screen. Similarly, an exhaust sound can accompany virtual “exhaust” of an object back into the virtual environment. The human player hearing a computer-generated suction sound creates the impression that the virtual suction tool stylus or other object has sucked up the object and now “contains” the object. The stylus or other object may virtually “store” any number of digital objects, the capacity being limited only by the game design.
In example illustrative non-limiting implementations, the human player may move the stylus to a different position in the game and expel the same (or transformed) object(s) back into the game environment. Accompanying exhaust sounds and graphical effects that make it appear as if the stylus is actually expelling them into the game environment. Alternatively or in combination, the object may be transformed by the virtual suction device (e.g., a red block may be transformed into fire) in the process of being virtually exhausted or expelled into the game environment.
The technology herein is thus capable of transforming an everyday common physical object such as an inert stylus into a device that may but does not necessarily need to exist in the real world. Exemplary illustrative non-limiting implementations thus provide methods and apparatus for a player to cross the boundary between the virtual world and the real world. Through displayed virtual world images and sounds, the player gets the impression that a virtual object can be brought into and manipulated in the real world. He similarly has the feeling that the physical object he is holding in his hand has virtual tool properties, functions and/or characteristics defining the manner in which it interacts with the virtual world. For example, in the case of a stylus being used as a virtual suction tool, the player may see the object distort and shrink as it is “sucked into” the stylus—even though the stylus is incapable of applying suction in the real world, is not connected to a vacuum pump or vacuum reservoir, and has no internal reservoir or other capacity for containing objects or material.
Accompanied by a suction-like sound, the player is given the impression that the plastic stylus has actually pulled and captured an object from the virtual world, bringing it into his world. If the player decides to empty the contents of the virtual suction tool back into the virtual world, an animation of the object being expelled from the stylus accompanied by a distinctive “exhaust” sound gives the player the impression that the object (or a transformed version of the object) has been returned from the real world back into the virtual world.
Particularly advantageous exemplary illustrative non-limiting implementations make use of a touch screen to provide the boundary between the real world and the 3D virtual world. A stylus, finger or other object interacting with a touch screen provides a convenient way to indicate position in the virtual world where the interaction is to occur. The fact that the stylus, finger or other object is directly in contact with the touch screen allows virtual world graphics at the point of contact to appear to be interacting directly with the physical object. Such an “Alice Through the Looking Glass” property of the virtual suction tool is also highly intuitive and natural and easy to use.
Other alternative arrangements using other pointing or position indicating arrangements, with or without distance between the object and the display, can also be provided. Thus, the physical object in one exemplary illustrative non-limiting implementation can be a touch screen stylus, but the physical object can be any object capable of interacting with a 3D or other graphics or other display system including but not limited to remote control devices, specially designed objects that have physical appearance of tools or weapons, etc. Since the physical object used for interaction is given virtual or simulated characteristics, capabilities and/or functions, the resulting virtual tool can differ from and/or exceed functions, capabilities and/or characteristics of actual physical objects existing in the real world. For example, a virtual suction tool can have properties that differ from and/or exceed the properties of a real world suction tool. An inexpensive, inert or other piece of plastic or other material can serve as a variety of different tools for a variety of games, each tool resembling, simulating or exceeding the bounds of any real world equivalent. The types and functions of each virtual tool are limited only by the imagination of the game developer. Because the game player has direct control over the virtual objects and can manipulate them seemingly in the real world, the player gains a high degree of control and a very interesting and engaging gaming experience. In a gaming context, the player feels as if he is directly manipulating the game object and so has a close connection to the game experience.
Further enhancements include expanding the type and range of objects the virtual suction tool can operate upon. For example, in some exemplary implementations, it is possible for the virtual suction tool to be used to move a typically rigid object such as a pipe, a girder, a conveyor belt, a ladder, a spring, a cannon, a hillside or any other item. Such items can represent real world or imaginary objects, and can have functions, behaviors and characteristics. The virtual suction device can apply suction to any of these items and thereby remove them from the virtual game environment. Then, the virtual suction device can expel previously suctioned items back into the game environment within guides such as dotted rectangular rectangles or boxes. The items can be elongated, and the orientation of the items can change between removal and reinsertion. For example, if the original item was an elongated pipe that was oriented horizontally and the guide into which the item is later inserted is elongated and vertically oriented, the reinserted elongated pipe will take on a vertical as opposed to horizontal orientation. Similarly, the item when reinserted can have its form and size adapt to the form and size of the insertion guide box so that the reinserted item can be larger or smaller than the original item before removal and can even have a different shape or form.
In one exemplary illustrative non-limiting implementation, the virtual suction device can remove multiple types of items from the game environment and keep track of the sequence in which different items were removed to provide a first-in-first-out or last-in-last-out order of reinsertion relative to removal. Thus, if the virtual suction device removes a pipe section and then a spring, the user might expect that it will first expel a pipe and then a spring, or first a spring and then a pipe. Or in some modes, the virtual suction device can provide a shuffle function to randomize the order of items expelled relative to absorbed. Or in some implementations, the game environment will determine which of several items in the virtual suction device's store or stack of previously-absorbed items is an appropriate one to expel upon the user touching the virtual suction device stylus to the touch screen.
The virtual suction tool can thus be used to dynamically change the game environment during play of a puzzle or other game to move items around, thereby allowing the game character to traverse the environment from one point to another. For example, the game character can pass through one pipe section to get to a second pipe section. One the character has passed through into the section pipe section, the user can use the virtual suction device to absorb the first pipe section and move it to a third position on the other side of the second pipe section. In this way, the user can reuse the same pipe section multiple times by moving it from one place to another in the virtual game environment.
In one exemplary illustrative implementation, each type of item has a guide or an empty place in the game environment, and items can be moved only to visual empty slots decided or placed to accept them (e.g., empty version or a pipe or empty version of a spring). There may be for example only two springs in a level and 20 empty spring slots. Part of the puzzle is to use the virtual suction device to move the two springs from one slot to another to allow the game character to traverse the gamescape and reach an objective or goal by moving a limited number of items from one preplaced location to another.
In one exemplary illustrative implementation, it is possible to move a small item to a large placement location for the item. If the user expels the previously sucked up item into a larger slot, the expelled item will fill the space. This is because the illustrative implementation treats the moving items as objects having behaviors (e.g., cannon can shoot, springs can compress and expand to launch characters, etc.) and not merely as buckets of paint or liquid.
In this example, some of the various items can be hollow and provide inlets and exits. Some of the items can be split items to provide additional flexibility. For example, a character can enter one pipe section and exit a different, unconnected pipe section.
Such techniques can also be used to create custom levels using an in-game editor. Thus, users can create customized game levels by specifying not only which items go where but also which item preplacement destination slots go where. Thus, the user can create custom puzzle levels that require other users to move items from one preplacement slot to another using the virtual suction device. The user creating the custom level can use the virtual suction device to pretest the level to ensure it functions. This provides the user with more freedom to solve the puzzle from moving objects from one location to another.